[OMNIML-6259] Speed up CI JIT compiles: Triton cache and arch-limited CUDA wheels - #2905

### What does this PR do?

Type of change: CI (job layout of the example tests)

Every example of the example workflow has its own job, and a job spends
a container start (156-252 s), the checkout and the base install before
its first test. Several examples run only 48-156 s of tests: llm_distill
102 s, llm_sparsity 156 s, specdec_bench 48 s and gpt-oss 54 s (1-GPU PR
run, 2026-10-21). This PR lets the runner take several examples and run
them one after the other in one job.

- **Runner (`_example_tests_runner.yml`):** `example` lists 1 to 3 names
separated by spaces (an empty list fails before any example runs). Each
example gets its own step (`Example 1: llm_eval`, `Example 2: gpt-oss`)
through a new composite action, `.github/actions/run-example`, that
installs the example's requirements, runs its tests with coverage and
uploads the coverage under the same `examples-<name>` flag as before
(Codecov's per-example carryforward is unchanged), except after a failed
run, allowed or not, whose partial coverage is no longer uploaded. A
failing example (its requirements install or its tests) does not fail
its own step: the action records the exit status and reports `failed` (a
failing step inside a composite action ends the job at once, whatever
`continue-on-error` the calling step has, as a hosted-runner check
showed, so the action uses none), so the next examples still run, as
when each had its own job with `fail-fast: false`, and the last step of
the job fails it. A job with one example fails in that last step too,
with an `Example failed` error annotation from the example's step; its
cache key is unchanged.
- **Requirements are installed right before each example's tests**, not
up front, so an example that upgrades packages does not change the
environment of the ones before it. The install and the tests run in one
step, and a failing install is a failure of its example like failing
tests, allowed or not by the list.
- **Allow-failure list:** the repo variable
`ALLOW_FAILURE_EXAMPLE_TESTS` (now `diffusers`) is passed to the runner
as `allow_failure_examples` and read per example inside the action (its
`Verdict` step: a warning for an allowed failure, an error annotation
and `failed=true` otherwise; whitespace in the variable is ignored, so
`diffusers, llm_qat` works); the callers lose their per-job
`contains(format(...))` expressions.
- **Trade-offs of sharing a job:** the grouped examples now share one
container, so a stray GPU-holding subprocess of an earlier example, or
caches and downloads piling up on one disk, can affect a later one, and
"Re-run failed jobs" re-runs every example of a group, not only the one
that failed (against about 900 runner-seconds saved per full PR run).
- **Grouped (`example_tests.yml`):** the torch lane runs `specdec_bench
llm_sparsity llm_distill` in one job (llm_distill last: it pins trl and
torchao), the trtllm lane runs `llm_eval gpt-oss` in one job (gpt-oss
last: it upgrades datasets 3.1.0 to 5.1.0 and trl). Both pairs share an
image, extras (`[hf,dev-test]`) and timeout, and their requirement files
do not conflict (read, and compared with the install logs of the
separate jobs: the three torch examples ended on the same transformers
5.18.0, peft 0.21.2 and accelerate 1.15.0, and in the trtllm pair only
gpt-oss changes anything, datasets and trl, which is why it goes last).
- **Not grouped:** llm_qat, speculative_decoding and hf_ptq (long jobs;
llm_qat at 1,362 s stays the longest job, so the workflow's wall time
does not grow), and **torch_trt into torch_onnx**: torch_trt's
requirements downgrade torch 2.13.0 to 2.12.1 and torchvision 0.28.0 to
0.27.1 (`torch-tensorrt<2.13`), so either the torch_onnx tests would run
on a different torch than they do, or the extension cache of the job
would hold builds from two torch versions. Its install step takes 204 s
and includes that torch swap, so grouping would save little anyway.

Lane gating (`pr-gate`), images, extras, timeouts and the
`example-pr-required-check` aggregation are unchanged: the jobs keep
their ids (`torch`, `trtllm`, ...), only their matrix entries and
display names change (for example `torch (specdec_bench llm_sparsity
llm_distill) / run-test`).

### Time improvement (estimated, then measured in this PR's run)

From the job step timings of the 1-GPU PR run of 2026-10-21 (container
init, checkout, install, tests, post):

| lane | jobs before | job seconds before | jobs after | job seconds
after (est.) | saved |
|---|---|---|---|---|---|
| torch (llm_distill, llm_sparsity, specdec_bench) | 3 | 342 + 378 + 282
= 1,002 | 1 | about 606 | about 396 |
| trtllm (llm_eval, gpt-oss) | 2 | 678 + 318 = 996 | 1 | about 726 |
about 270 |

13 example jobs become 10: about 660 runner-seconds less per full PR run
(about 8% of the roughly 7,800 job-seconds of a full run; the nightly
saves the same container starts on 2-GPU runners). The grouped jobs keep
their tests and installs; what disappears is the container start
(156-252 s), checkout, caches and the base install of the second and
third job. The merged jobs (about 606 and 726 s) are shorter than the
workflow's longest job, so the wall time does not change. Measured in
this PR's own run (run 39686590762, 1-GPU runners, every lane green):

- torch: the merged job (`specdec_bench llm_sparsity llm_distill`) took
519 s (container 182 s, install 29 s, the three examples 54 + 149 + 91
s). The three separate jobs of a run of the same day took 273 + 342 +
948 = 1,563 s, but the 948 included an unusually slow 649 s container
start (about 190 s is typical), so about 1,104 s typical: the merge
saves about 585 s (estimate: 396 s).
- trtllm: the first run of the merged job (`llm_eval gpt-oss`) took
1,271 s because its cache key is new and the extension cache was cold
(install 204 s instead of about 30 s, llm_eval 553 s instead of about
260 s). A re-run of that job with the cache saved took 600 s (container
220 s, install 20 s, llm_eval 285 s, gpt-oss 61 s), against 572 + 351 =
923 s for the two separate jobs of a run of the same day: about 323 s
saved (estimate: 270 s). After this PR lands, the first run of each
grouped job on main pays the cold build once.
- Together about 908 runner-seconds less per full PR run (estimate:
660); the wall time does not change, llm_qat (1,260-1,556 s) and hf_ptq
(up to 1,764 s) stay the longest jobs.

### Usage

Nothing user-facing. To group another small example, list its name in
the matrix entry of its lane (`example: ["a b", c]`), put the example
that pins or upgrades packages last, and keep at most 3 names per job.

### Testing

- `actionlint` 1.7.7: clean on all workflows (it caught that expressions
have no `replace()`, so the cache key is built in a shell step; for a
single example it is identical to the old key).
- PyYAML comparison of the old and new `example_tests.yml`: every
example is still run exactly once, images, extras and timeouts per lane
are equal, `pr-gate` and `example-pr-required-check` are equal, 13 jobs
become 10.
- Stub test of the new shell step with one, two, three and four examples
(four fails loudly).
- pre-commit on the range (yamlfmt reformatted the new action file once,
then passed).
- Run on GitHub runners: this PR changes `example_tests.yml`, which
every lane's change filter includes, so its own CI ran all lanes (all
required checks green on 3048746, ab0a36f and e332c2e, and,
with the install change below, on 5d6dfe5, the head of this PR;
numbers above): the two grouped jobs and the single-example jobs through
the new step path. Those runs built the extension cache cold under the
new group keys (the 1,271 s first run of the trtllm group) and ran the
per-example Codecov upload steps: every example uploaded its coverage
(one `Upload coverage to Codecov` step per example in each job log). No
example failed in them, so the failure paths were run separately (next
bullet).
- Failure paths on a hosted runner (a throwaway branch with a
push-triggered workflow, deleted afterwards; no GPU): the real action
and the real example steps, cut out of `_example_tests_runner.yml`, with
a passing and a failing fake example in six scenarios (an allowed or a
blocking failure in a one-example job, as the first and as the last of
two). First run (run 39821769250, this PR before e332c2e, with
`continue-on-error` on the example steps): an allowed failure left the
job green with the warning and no coverage upload for the failed
example, but a blocking failure in the first example skipped the second,
because a failing step inside a composite action ends the job at once
whatever `continue-on-error` the calling step has. The action therefore
no longer fails on test failures: it reports `failed`, and the last step
of the job fails it. Second run (run 39822372305, e332c2e): all six
scenarios behave as designed. Third run (run 39828420032, this version,
which also records the install's exit status and drops
`continue-on-error` from the action): eleven scenarios, the six above
plus a requirements install that cannot resolve (blocking as the first
of two, and allowed), an allow list written `fake_pass, fake_fail`, the
same split over several lines, and an entry that only shares a prefix
with the example. All behave as designed: allowed failures, including a
failing install, leave the job green with the warning and no coverage
upload for the failed example; blocking failures add an `Example failed`
error annotation, still run the remaining examples (the passing one
uploads its coverage, and an install failure in the first example does
not stop the second) and fail the job in the final step; the prefix does
not match.

### Before your PR is "*Ready for review*"

Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).

Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.).

- Is this change backward compatible?: N/A (CI only; the runner's
`allow_failure` input became `allow_failure_examples`, and its only
caller is `example_tests.yml`)
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: no new
dependency
- Did you write any new necessary tests?: N/A (workflow change; the PR's
own run exercises it)
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A (CI only)
- Did you get Claude approval on this PR?: ✅ (`/claude review` approved
e332c2e, with three non-blocking suggestions)

### Additional Information

Part of the GPU-CI cost work under OMNIML-6177 (task OMNIML-6180,
example lanes). It edits `_example_tests_runner.yml`, as the open cache
PR #2905 does (this PR touches the cache key line, #2905 the cache
steps, so expect a small conflict); whichever lands second rebases. The
repo variable `ALLOW_FAILURE_EXAMPLE_TESTS` needs no change.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Tests**
* CI can run up to three selected examples sequentially in a shared job.
Later examples still run if an earlier example fails, and the job
reports any failures that are not allowed.
* Failures can be allowed for specific examples, so they do not count as
unallowed failures.
* Coverage is uploaded separately for each example when installation and
tests succeed.
  * Example-specific setup is handled independently for each run.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>